We present a game mechanic called pseudo-visibility for games inhabited by non-player characters (NPCs) driven by reinforcement learning (RL). NPCs are incentivized to pretend they cannot see pseudo-visible players: the training environment simulates an NPC to determine how the NPC would act if the pseudo-visible player were invisible, and penalizes the NPC for acting differently. NPCs are thereby trained to selectively ignore pseudo-visible players, except when they judge that the reaction penalty is an acceptable tradeoff (e.g., a guard might accept (...) the penalty in order to protect a treasure because losing the treasure would hurt even more). We describe an RL agent transformation which allows RL agents that would not otherwise do so to perform some limited self-reflection to learn the training environments in question. (shrink)

After generalizing the Archimedean property of real numbers in such a way as to make it adaptable to non-numeric structures, we demonstrate that the real numbers cannot be used to accurately measure non-Archimedean structures. We argue that, since an agent with Artificial General Intelligence (AGI) should have no problem engaging in tasks that inherently involve non-Archimedean rewards, and since traditional reinforcement learning rewards are real numbers, therefore traditional reinforcement learning probably will not lead to AGI. We indicate two possible ways (...) traditional reinforcement learning could be altered to remove this roadblock. (shrink)

Can an agent's intelligence level be negative? We extend the Legg-Hutter agent-environment framework to include punishments and argue for an affirmative answer to that question. We show that if the background encodings and Universal Turing Machine (UTM) admit certain Kolmogorov complexity symmetries, then the resulting Legg-Hutter intelligence measure is symmetric about the origin. In particular, this implies reward-ignoring agents have Legg-Hutter intelligence 0 according to such UTMs.

Inspired by recent progress in multi-agent Reinforcement Learning (RL), in this work we examine the collective intelligent behaviour of theoretical universal agents by introducing a weighted mixture operation. Given a weighted set of agents, their weighted mixture is a new agent whose expected total reward in any environment is the corresponding weighted average of the original agents' expected total rewards in that environment. Thus, if RL agent intelligence is quantified in terms of performance across environments, the weighted mixture's intelligence is (...) the weighted average of the original agents' intelligences. This operation enables various interesting new theorems that shed light on the geometry of RL agent intelligence, namely: results about symmetries, convex agent-sets, and local extrema. We also show that any RL agent intelligence measure based on average performance across environments, subject to certain weak technical conditions, is identical (up to a constant factor) to performance within a single environment dependent on said intelligence measure. (shrink)

Fitch's Paradox and the Paradox of the Knower both make use of the Factivity Principle. The latter also makes use of a second principle, namely the Knowledge-of-Factivity Principle. Both the principle of factivity and the knowledge thereof have been the subject of various discussions, often in conjunction with a third principle known as Closure. In this paper, we examine the well-known Surprise Examination paradox considering both the principles on which this paradox rests and some formal characterisations of the surprise notion, (...) crucial in this paradox. Standard formalizations of the Surprise Examination paradox in modal logic do not seem, at first glance, to depend on either factivity or knowledge-of-factivity, but we will argue that both factivity and knowledge-of-factivity play a key implicit role in the paradox. Namely, they are implicitly, perhaps unintentionally, used in order to simplify the definition of surprise. We analyze modal logical formalizations of three versions of the paradox concluding that the Surprise Examination paradox is the result of two flaws: the assumption of knowledge-of-factivity, and the over-simplification of the definition of "surprise" accordingly. By fixing these two flaws, the Surprise Examination paradox vanishes. (shrink)